Estimating Classifier Performance in Unknown Noise

We propose and investigate a non-parametric method for identifying regions of speech that have unexpected distortions not seen in the training data. The method does not require knowledge of correct labels and relies only on divergence between statistics of the test and training data. Our experiments show that the proposed method requires a relatively small amount of test data of the order of several seconds to stabilize, and correlates well with recognition error observed on the test data.